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Visual  Knowledge  in  Tactical  Planning: 
Preliminarj-  Knowledge  Acquisition 
Phase  I  Technical  Report 


Introduction 

Program  Goals  and  Motivation 

Over  the  past  twenty  years,  much  has  been  learned  about  the  structure 
and  use  of  human  knowledge  and  the  potential  values  of  computer 
representation  and  manipulation  of  knowledge  and  reasoning  for  a  variety  of 
problem  solving  purposes.  However,  very  little  of  this  work  has  concerned 
itself  with  those  components  of  human  knowledge  that  are  not  easily  expressed 
in  verbal  terms,  for  reasons  both  of  difficulty-  in  assessing  the  knowledge 
involved  and  of  representing  it  within  the  confines  of  a  computer 
representation.  Similarly,  computing  capabilities  in  image  understanding  and 
in  graphics  displays  have  advanced  considerably,  but  little  has  been 
accomplished  in  terms  of  reasoning  with  or  about  images  themselves.  TTte 
work  described  here  is  part  of  a  project  focused  on  the  non-verbal  (visual) 
components  of  knowledge  used  in  a  map-based  planning  task.  This  phase  of 
the  project  involved  knowledge  acquisition  activities  oriented  toward 
identifying  the  visual  objects  and  features  of  objects  that  are  of  tactical 
importance.  The  long-term  intent  of  the  work  is  to  combine  the  verbal 
components  of  the  task  and  required  knowledge  with  the  visual  components  to 
generate  a  user  interface  for  a  tactical  planning  decision  support  system  whose 
displays  and  behavior  are  compatible  in  form  and  content  with  the  cognitive 
representations  of  its  users  and  whose  internal  representations  support  direct 
manipulation  of  both  visual  and  verbal  knowledge.  To  fully  investigate  this 
area,  the  project  will  draw  and  build  on  work  in  three  major  research  areas, 
each  of  which  is  discussed  briefly  in  the  following  sections. 

Knowledge-based  Planning 

Planning  and  problem  solving  are  the  primary  areas  of  reasoning 
addressed  by  “intelligent”  or  knowledge  based  systems.  Planning  is  generally 
viewed  as  the  process  of  finding  a  series  of  actions  that  will  eliminate 
differences  between  the  initial  state  and  some  goal  state  (Hammond,  1984; 
Minsky,  1975).  Numerous  techniques  have  been  developed  and  used 
successfully  for  various  planning  situations  including:  building  plans  for  every 
situation  from  rules  (Sacerdoti,  1975);  searching  for  complete  (or  fixable) 
plans  from  within  an  index  of  existing  plans  (Hammond,  1984);  and  reasoning 
from  earlier  cases  or  experiences  (Kolodner,  1985).  Applications  have  been 
built  that  deal  with  planning  domains  and  problems  ranging  from  extremely 
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simple  “toy”  domains  with  few  or  no  interactions  between  goals  to  highly 
complex  domains  with  conflicting  or  competing  goals.  Research  and 
development  efforts  in  knowledge  based  planning  have  been  highly  successful 
and  several  large-scale,  complex  planning  systems  (such  as  DARPA’s  Pilot’s 
Associate  and  AirLand  Battle  Management  FORCES  planner)  have  been 
prototyped.  These  systems  derive  complex  plans  in  short  time-frames  and, 
when  necessary,  use  different  planning  techniques  depending  on  the 
knowledge  and  reasoning  demands  of  the  task. 

One  thing  ail  these  approaches  have  in  common  is  that  the  domains  in 
which  the  plans  are  generated  (or  the  problems  for  which  plans  must  be  made) 
are  expressed  solely  in  semantic  terms:  there  are  no  illustrations  or  diagrams 
provided.  All  information  provided  is  in  some  verbal  form.  In  human 
experience,  however,  we  rarely  encounter  a  situation  that  provides  only 
semantic  or  verbal  input.  Instead,  any  situation  perceived  provides 
information  in  multiple  modalities  and,  in  fact,  we  may  rely  on  a  non-verbal 
mode  for  much  of  our  understanding  of  the  situation  (Anderson.  1984;  Larkin 
&  Simon,  19L87).  In  particular,  much  of  what  we  perceive  and  manipulate  in 
day-to-day  planning  is  visual  in  nature,  and,  in  some  tasks,  almost  all  the 
pertinent  information  is  provided  via  diagrams,  maps,  and  other  illustrations. 

Visual  Knowledge 

Visual  experience  forms  a  major  pan  of  our  total  fund  of  knowledge 
(Chipman,  1987).  The  knowledge  we  gain  and  retain  from  visual  experience  is 
often  itself  imagistic  or  visual  in  nature  and  differs  from  verbal  or  semantic 
knowledge  in  form,  content,  and  style  of  reasoning.  In  other  words,  visual 
knowledge  is  the  internal  representation  of  our  visual  experience  and  differs 
from  verbal  knowledge  just  as  visual  experience  differs  from  linguistic 
experience.  However,  to  date,  no  theoretical  model  has  been  developed  that 
adequately  captures  the  nature  of  visual  knowledge  or  the  relationships  and 
interactions  of  verbal  and  visual  knowledge. 

By  visual  knowledge,  we  refer  to  that  part  of  human  knowledge  that  is 
difficult  to  express  in  spoken  or  written  language,  but  easy  to  communicate 
through  pictorial  means  such  as  photographs,  diagrams,  chans,  and  graphs. 
This  includes,  but  is  not  limited  to,  information  about  the  details  of 
appearances  of  objects  in  a  scene,  the  spatial  relationships  among  objects  in  a 
scene,  and  specific  characteristics  or  features  of  individual  or  collective 
objects.  Visual  knowledge  can  be  considered  in  contrast  to  verbal 
knowledge — that  part  of  human  knowledge  that  is  easily  communicated  in 
spoken  or  written  language,  such  as  information  about  the  names  or  number  of 
items,  concepts,  actions,  and  relationships  among  actions.  While  substantial 
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evidence  has  been  obtained  supporting  the  existence  of  visuaily-based  internal 
knowledge  representations  (Anderson,  1984;  Pavio.  1986;  Vekker,  1974)  and 
demonstrating  the  value  of  external  graphic  representations  in  supporting 
reasoning  (Centner,  1989;  Larkin  &  Simon,  1987),  no  one  has  developed  a 
sufficient  model  for  the  organization  and  content  of  internal  images  and  their 
manipulation  during  reasoning  processes.  This  lack  of  a  theoretical  mode!  of 
the  cognitive  representation  and  manipulation  of  visual  knowledge  restricts  our 
capability  to  fully  utilize  the  graphic  display  and  image  processing  capabilities 
of  existing  computer  systems  in  several  critical  w'ays,  particularly: 

•  imparting  knowledge  for  training  purposes; 

•  representing  imaeistic  information  in  computer  storage;  and 

•  supporting  visually  based  communication  between  human  and 

machine. 

Image  Understanding  Systems 

Image  understanding  systems  have  also  developed  dramatically  in 
recent  years.  In  fact,  several  “knowledge-based”  systems  have  been 
developed  (Draper,  et.  al.,  1989;  Gilmore  &  Shapiro,  1988).  However,  in 
almost  all  these  systems,  the  application  of  KB  processing  to  image 
understanding  occurs  only  at  a  high  level,  after  the  input  image  has  been 
mapped  (Gilmore  &  Shapiro,  1988).  In  addition,  the  most  developed  uses  of 
such  systems  are  primarily  oriented  toward  identification  of  objects  and 
extraction  of  semantic  information  providing  the  name  and  location(s)  of  each 
object.  Consequently,  in  current  applications,  additional  reasoning  done 
relative  to  the  image  is  based  on  the  semantic  information  extracted  from  the 
image.  Brady  (1987)  lists  ten  such  areas  in  intelligent  vision  that  are 
sufficiently  mature  for  applied  use.  He  places  direct  reasoning  about  images 
and  integration  of  computer  vision  systems  with  other  techniques  in  a  class  of 
areas  of  research  interest  likely  to  pay  off  in  the  near  future. 

With  this,  however,  we  return  again  to  the  evidence  that  much  of 
human  reasoning  involves  the  direct  perception  and  manipulation  of  visual  or 
imagistic  information,  rather  that  translation  of  that  information  to  semantic 
form  for  reasoning.  Each  of  the  technology  areas  discussed  in  the  previous 
sections  has  begun  to  consider  the  problem  of  how  humans  manipulate  visual 
information  and/or  how  intelligent  systems  should  act  on  images,  but  none  has 
yet  structured  an  approach  to  mimicing  human  behavior  in  a  knowledge  based 
system.  It  is  the  goal  of  this  project  to  address  this  critical  issue. 
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Methodology 

For  the  current  project,  the  domain  of  interest  is  military  tactical 
planning  at  the  brigade  level.  ..\bove  this  level,  particularly  at  echelons  of 
corps  and  above,  planning  is  not  tightly  bound  to  the  specifics  of  terrain  as 
portrayed  on  a  map.  Planners  at  such  levels  are  concerned  with  broad 
expanses  of  terrain  and  generally  work  with  maps  at  levels  of  detail 
insufficient  to  support  actual  reasoning  about  specific  terrain  objects  and  their 
relationshios.  Planning  at  the  brigade  level  is  done  using  maps  of  sufficient 
detail  to  allow  consideration  of  individual  terrain  features  and  is  concerned 
with  the  placement  of  units  small  enough  for  their  performance  to  be  affected 
by  such  terrain  units. 

Materials 

Domain  experts  were  provided  with  a  1:50,000  scale  map  of  the  Fulda 
region  of  Germany  and  an  operations  plan  for  defensive  actions  by  the  2nd 
Brigade  of  the  23rd  Armored  Division.  The  operations  plan  was  taken  from  US 
Army  subcourse  313/4-Defensive  Tactics,  Lesson  4-Brigade  Defensive 
Planning.  The  operations  plan  included  the  brigade  order  of  battle  and  task 
force  organization  as  specified  by  the  division  commander  and  ail  major 
paragraphs  required  in  an  OPLAN,  but  no  annexes.  Sufficient  map  sketches 
were  included  to  locate  brigade  boundaries,  task  force  boundaries,  FEBA,  and 
critical  defensive  positions.  For  the  counterattack  and  full  offense  problem 
solving  sessions,  the  same  geographic  location  and  order  of  battle  were  used, 
but  no  operations  plans  were  available. 

Participants 

Two  former  Army  officers  participated  as  domain  experts.  One  (BB) 
has  25  years  of  experience  in  the  US  Army  with  command  experience  up  to 
the  brigade  level.  In  addition,  he  served  one  tour  as  a  tactics  instructor  at  the 
Armor  school.  The  other  expert  (RP)  has  20  years  of  experience  in  the  US 
Army,  primarily  as  an  engineer  and  is  a  graduate  of  the  US  Army  Command 
and  General  Staff  College.  Both  experts  were  familiar  with  the  European  area 
used  in  this  study  and  with  both  US  and  Soviet  doctrine  and  tactics  for  fighting 
in  that  area. 

Procedure 

Seven  knowledge  engineering  sessions  were  held  with  BB  and  six  with 
RP  for  a  total  of  approximately  seven  hours  of  interactions.  The  first  session 
with  each  expert  was  a  semi-structured  interview  covering  basics  of  military 
planning  and  determining  what  information  and  materials  would  be  needed  to 
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support  the  subsequent  problem  solving  sessions.  The  first  session  was 
recorded  on  audio  tape  and  transcribed  tor  analysis.  The  problem  solving 
tasks  for  subsequent  sessions  were  presented  in  the  following  order: 

•  general  terrain  analysis — primarily  from  Blue  perspective; 

•  terrain  analysis  from  Red  (offensive)  perspective; 

•  position  Blue  troops  for  defense; 

•  “play  back”  one  battalion  sector  of  defense; 

•  position  troops  for  limited  counterattack  by  brigade;  and 

•  position  troops  for  division  offense  from  Fulda  river  to  regain 

international  border 

Results 

All  problem  solving  sessions  were  recorded  on  video  and  audio  tape. 
The  audio  tapes  were  transcribed  for  coding.  The  initial  coding  scheme  was 
established  to  extract  from  the  transcripts  terms  of  reference  for  and  some 
relationships' between  visual  features  of  the  map  itself  and  the  terrain  features 
of  importance  to  the  planners  reasoning  processes.  Specifically,  the 
transcripts  have  been  coded  for  visual  map  characteristics  that  identify 
features  of  terrain  units,  specific  and  composite  terrain  units,  features  of 
specific  terrain  units  that  are  of  tactical  importance,  and  roles  of  terrain  units 
in  defensive  and  offensive  planning.  All  coding  has  been  done  by  the  principal 
investigator. 

During  their  planning  sessions,  the  experts  referred  to  five  types  of  map 
characteristics:  specific  symbols  that  are  defined  in  the  map  legend  such  as 
those  for  deciduous  and  coniferous  trees;  map  coloration  such  as  green  for 
forested  areas  and  brown  for  cleared  areas;  contour  lines  indicating  slope  of 
the  land;  absolute  locations  of  terrain  features;  and  relative  locations  of  terrain 
features.  Each  of  these  was  vital  to  the  identification  and  consideration  of  the 
terrain  units  discussed.  The  two  experts  mentioned  a  total  of  13  man-made 
terrain  units  (roads,  towns,  bridges),  22  naturally  occurring  specific  terrain 
units  (hill,  river,  stream),  and  11  composite  terrain  units  (avenue  of  approach, 
field  of  fire).  Composite  terrain  units  are  formed  based  on  the  relationships 
among  specific  units. 

The  coded  information  from  the  transcripts  has  been  structured  into  a 
knowledge  document  and  is  attached  to  this  report  as  Appendix  A.  Further 
analysis  of  the  transcripts  and  of  the  video  tapes  will  continue  during  Phase  II 
of  the  project. 
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Program  Accomplishments 

As  proposed,  Phase  I  of  this  project  has  focused  on  selecting  an 
appropriate  planning  problem  for  knowledge  acquisition  and  completing  initial 
kn'"  - ledge  acquisition  efforts.  From  the  elicited  transcripts  and  initial  coding, 
the  project  thus  far  has  succeeded  in: 

•  establishing  the  immediacy  and  importance  of  visual  information  in 
tactical  planning; 

•  identifying  a  set  of  terrain  objects  and  their  features  and  roles  that 
are  critical  in  tactical  considerations; 

•  identifying  a  set  of  map  characteristics  that  inform  the  perceiver 
about  the  terrain  objects  and  their  features; 

•  identifying  a  set  of  composite  terrain  features  that  are  important  in 
tactical  planning;  and 

•  capturing  substantial  evidence  of  the  types  of  reasoning  applied  to 
visual  information  during  tactical  planning. 

Phase  II  Goals 

During  the  second  phase  of  this  project,  several  subgoals  will  need  to  be 
met  in  order  to  complete  Phase  II  with  adequate  specification  of  the  knowledge 
representation  and  functional  capabilities  required  by  a  user  interface  to 
support  map-based  tactical  planning.  Briefly,  these  subgoals  are: 

•  complete  analysis  of  video  tapes  with  a  focus  on  the  reasoning  or 
mental  operations  applied  to  map  characteristics  and  terrain  objects; 

•  funher  knowledge  engineering  sessions  to  focus  on  the  relationsh-ps 
between  terrain  objects  that  identify  composite  terrain  features; 

•  develop  a  statement  of  the  functional  capabilities  needed  by  a  user 
interface  capable  of  supporting  such  reasoning; 

•  review  the  types  of  data  and  knowledge  representations  currently 
used  or  generated  by  knowledge  based  planning  and  vision  systems; 

•  select  and  revise  an  appropriate  representation; 

•  develop  functional  specifications  for  an  interface;  and 

•  begin  prototyping  efforts. 
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Appendix  A:  Preliminary  Dictionary  of  Map  Objects 


4)  on 

c 

s:  -z: 


•o  to 

a:  ^ 


o  u 
o  o 
U 


.2  "S 

O  Q. 

or) 


x:  ^ 

•=  «3 

•o  a. 

^  U 


t/5 

C  w 

-  8  S 

1^1 

<u  5?' 

>  2  Jr 

O  E  ® 


c 

o 

>> 

k- 

*1^ 

u 

X) 

c 

w. 

3 

O 

OJ 

«n» 

o 

o 

> 

c 

VI 

ea 

V3 

o 

JC 

a. 

u 

o 

•o 

'W 

XJ 

3 

o 

■2 

IE 

•a 

o 

JC 

V) 

o 

o 

w 

a. 

V5 

c 

>x 

c 

o 

(Aw 

<u 

Q 

.2  O 

An'  4i 

0)  c. 


w  C3 

t3  C. 
^  C3 

^  u 


C  T5 

.2  ^ 

o>  a. 
«  00 


b  >;  .2 

.r:  03 

•B  ^  o 

03  Cl  ^ 

■■era  <u 

^  O  ra 


Candy-Stripe  |  Width,  Direction  |  Color  (R/W  alternating) 

j  Capacity,  Speed  | 


.bi  tJ 
^  c. 

a  </2 


•w  C3 

T3  CX 
73 

^  U 


W 

.*  ^ 
U  Q. 

ST 


O  2 
ti.  (/3 


O 

.2 

2  Q. 

a  on 
Q  ^ 

-c  S 

T3  C. 
ir  73 

^  U 


.K  o 
o  ^ 
e  V5 


-c  2 

^  n 
-3  C. 

^  U 


>  S  o 
<  E  b 


o  u 
w  £  'S 

V>  ^  V7 

3  ^  U 

a>  u. 

^  o* 

£  <«  w 

.2  •=„  « 

ir  oo  > 

o  x:  ii 

Q  </.  «" 

til  w  ^ 
Qi  O  = 
“•  o.  C3 
w  Q.  3 

0  3"’ 

U,  WO  D 


ci> 

c 

</". 

c. 

cr. 

o 

P3 

-o 

JO 

U 

a 

o 

r3 

E 

t/O 

c 

u« 

C 

c 

o 

s 

c. 

u. 

CJ 

L. 

— 

'5 

-  ‘ 

■ 

u  o. 
V.  O 


<L>  **>• 

tJ  o 

W  L. 

o  o 

Um  llm 


If  dense,  supports  movement  options 
If  sparse  and  terrain  vvoodcd,  limits  movement 

If  sparse  and  terrain  level  anti  open,  allows  cross  country  movement 


For  defense,  good  terrain 

For  defense,  supports  fields  of  fire 

Usually  elevated  on  man-made  embankment 
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Mountain;  Hill  !  Height.  Rel.  Height  1  Numerical  indicator 

I  Slope.  Roads  j  Density  of  contour  lines 

I  Vegetation.  Visibility  j  Color  (Green  or  Brown) 


Water  Course 


Provides  coneeaimem 
if  (hick,  inhibits  movement 
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Any  constricting  terrain 

F'or  defense,  important  to  control 

F'or  defense,  usually  easy  to  control 


Poor  terrain  for  offense,  indicates  location  where  defense  can  stop  movment 


